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PREFACE 

This  thesis  is  concerned  with  the  application  of  statistical  methods 
to  naval  operational  testing,  and  more  specifically,   with  testing  of  the 
type  conducted  by  OpDevFor  and  similar  testing  agencies.  *  The  statis- 
tical methods  considered  are  those  of  confidence  limits,    sequential 
analysis  and  the  more  recently  developed  Statistical  Decision  Theory. 
These  techniques  are  regarded  as  of  interest  to  Naval  line  officers  in 
various  categories  of  billets,   particularly  Project  Officers  at  testing 
agencies  and  officers  concerned  with  planning  which  is  based  on  the 
results  of  testing  programs.     Frequently,    such  officers  are  unfamiliar 
with  the  use  and  limitations  of  the  three  methods  and  the  relations  of 
these  methods  to  each  other. 

Statistical  Decision  Theory,    in  particular,    can  be  useful  at  higher 
levels  of  the  naval  establishment  such  as  the  offices  of  CNO.     Naval 
planners  at  this  level  may  be  faced  with  a  difficult  problem  in  connec- 
tion with  testing  programs  whose  objective  is  the  estimation  of  the  per- 
centage effectiveness  or  probability  of  success  of  a  weapon,   in  future 
combat.     Such  programs  are  costly  to  conduct,   and  increasingly  so 
when  an  expensive  weapon  is  tested  to  destruction.     Costs  of  a  different 
nature  are  those  associated  with  the  possible  consequences  if  a  poor 
estimate  of  the  weapon's  effectiveness  is  obtained  from  tests.      The  fun- 
damental problem  is  to  determine  how  many  trials  are  to  be  conducted 
and  hence  how  many  weapons  snould  be  tested.     Attempts  to  solve  the 
problem  by  reconciling  the  conflicting  costs  will  generally  lead  to  a 
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dilemma..     The  application  of  Statistical  Decision  Theory  to  this  problem 
is  contingent  upon  the  ability  of  the  planner  to  specify  the  inputs  or  data 
required  by  the  theory.      An  essential  objective  of  this  thesis  is  to  show 
how  a  planner  might  be  guided  in  specifying  these  inputs. 

The  thesis  has  been  written  with  a  view  towards  its  usefulness  for 
personnel  with  a  minimum  background  in  probability  and  statistics.  It 
is  addressed  also  to  students  and  practicioners  of  Operations  Analysis 
who  may  be  concerned  with  the  relations  of  the  three  statistical  methods 
which  are  considered  as  tools  which  may  provide  quantitative  basis  for 
executive  decision. 

The  writer's  interest  in  the  possible  applications  of  this  theory  was 
aroused  during  the  study  of  Statistical  Decision  Theory  at  the  U.   S. 
Naval  Postgraduate  School.     The  need  for  investigations  as  to  how  the 
inputs  could  be  specified  was  pointed  out  by  L.T".   R.    A.    Tucker,   USN  in 
his  thesis:    An  Introduction  to  Statistical  Decision  Functions  [  1  ] . 
Tucker's  paper  presents  detailed  discussions  of  the  mathematical  con- 
cepts involved  in  the  theory  and  the  precise  mathematical  steps 
required  to  obtain  the  solution.     It  is  intended  for  readers  with  less 
mathematical  background  than  is  required  for  an  understanding  of  the 
basic  work  Statistical  Decision  Functions  by  Abraham  Wald  [2]  . 
Readers  who  are  interested  in  the  theory  and  detailed  computations 
should  refer  to  Wald  and  Tucker. 

This  paper  is  divided  into  five  chapters.     Chapter  I  discusses  the 
relationship  between  types  of  operational  testing  problems  at  various 
levels  of  the  Navy.     Chapters    II  and  III  are  examples  of  applications 
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of  statistical  methods  at  the  testing  agency  level.     In  Chapter    IV,   an 
example  of  a  guided  missile  is  used  as  a  vehicle  of  discussion  as  to  how 
the  inputs  required  by  Statistical  Decision  Theory  may  be  specified  by 
an  office  of  CNO.      A  solution  to  the  example  is  then  given.     In  Chapter 
V   the  effect  of  variation  of  parameters  is  shown. 

This  thesis  was  written  at  the  U.    S     Naval  Po-stgraduate  School, 
Monterey,   California,   during  the  period  January-May,    1956.      lam 
indebted  to  Professor  Thomas  E.    Oberbeck  for  his  continued  patience, 
encouragement  and  most  capable  guidance  while  acting  as  faculty  advi- 
sor;   and  for  permission  to  use  solutions  he  has  obtained  by  program- 
ming the  problem  for  an  electronic  computer.     I  wish  to  thank  Professor 

C.  A.    Magwire  for  his  valuable  assistance  as  second  reader,  and  Mrs. 

D.  P.    Slingerland  for  her  meticulous  preparation  of  this  typescript. 
Appreciation  is  also  expressed  here  to  personnel  of  VX-4  and  the  U.    S. 
Naval  Air  Missile  Test  Center,    Point  Mugu,    California,   who  provided 
much  helpful  information  on  the  practical  aspects  of  testing. 

The  graph  on  page     8     is  reproduced  from  Burrington  and  May's: 
Handbook  of  Probability  and  Statistics  by  permission  of  the  publishers, 
Handbook  Publishers,   Inc.  ,   Sandusky,    Ohio. 
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CHAPTER      I 
INTRODUCTION 

This  thesis  is  concerned  with  the  application  of  some  statistical 
methods  to  testing  programs  of  the  type  conducted  by  OpDevFor  or 
similar  agencies,   operating  directly  for  CNO.     Such  programs  are 
defined  as  naval  operational  testing.     The  thesis  does  not  consider 
applications  of  these  .methods  to  quality  control  or  to  development  or 
engineering  tests  such  as  those  which  might  be  conducted  by  the  mater- 
ial Bureaus. 

The  scope  is  further  limited  to  those  operational  testing  pro- 
grams in  which  the  object  is  to  obtain  an  estimate  of  the  probability 
of  success  of  a  weapon  or  weapon  system,  in  future    combat.     As  the 
phrase  is  used,  probability  of  success  can  be  thought  of  as  generally 
equivalent  to  hit  probability,    percent  effectiveness  or  reliability.     The 
term  testing  program  is  used  to  describe  any  test  which  consists  of  a 
series  of  independent  trials  conducted  to  obtain  the  above  estimate. 
The  estimate  is  regarded  as  providing  a  quantitative  basis  for  execu- 
tive decision  at  some  level  of  the  Naval  organization. 

A  fundamental  problem  in  any  testing,  program  is  the  determination 
of  the  number  of  trials  to  be  conducted.     For  some  programs  this  can 
be  difficult'  ,   depending  essentially  on  the  magnitude  of  various  "costs". 
It  is  clear  that  the  number  of  trials  is  related  to  cost.     The  reconcilia- 
tion of  all  the  costs  involved  is  sometimes  difficult.     This  concept  is 
well  phrased  by  Breakwell  [3]    .     In  a  sentence  taken  out  of  content: 
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A  balance  is  sought  between  (1)    the  cost  of  testing    for  reliability 
and    (2)    the  risks,   because  of  limiting  testing,    of  either  accepting 
an  insufficiently  reliable  product  or  rejecting  a  sufficiently  reli- 
able one. 

As    (1)   increases  the  natural  tendency  is  toward  fewer  trials;  as  (2) 
increases  the  tendency  is  to  desire  more  trials.     Thus,   if  a  balance  is 
to  be  obtained,    the  statistical  methods  used  must  explicitly  relate  the 
"cost",   the  number  of  trials  and  the  decisions  to  accept  the  product  or 
weapon.     Statistical  decision  theory  provides  a  rational  basis  for  attack- 
ing this  problem  but  it  can  only  be  applied  when  these  costs  can  be  spe- 
cified.    Frequently,   these  costs  cannot  be  estimated  by  the  testing 
agency.     Under  these  circumstances,   the  costs  must  be  furnished  to 
the  agency  or  the  agency  is  compelled  to  apply  a  statistical  method  which 
is  not  based  on  such  estimates.     Chapters    II  and   III  are  devoted  to  such 
methods. 

Line  officers  who  are  unfamiliar  with  statistical  techniques  may 
often  be  directly  concerned  with  testing  programs.     Chapter   II  serves 
to  introduce  the  technique  of  confidence  intervals,    which  are  often  used 
in  reporting  test  results.     It  points  out  that  this  technique  does  not  pro- 
vide the  planner  of  the  testing  program  with  adequate  guidance  as  to 
how  he  should  specify  the  numbe  r  of  trials.     Chapter    III   illustrates 
the  use  of  sequential  analysis.     It  is  considered  applicable  to  testing 
programs  which  are  conducted  to  compare  improved  weapons  with  an 
existing  weapon. 

Testing  programs  which  involve  new  weapons  such  as  guided  mis- 
siles,  where  the  costs  in   (1)  and   (2)  are  high,   provide  a  possible  field 
for  application  of     statistical  decision  theory.      It  is  considered  that  the 


cost  estimates  required  may  be  available  to  naval  planners  at  the  CNO 
level.     Chapter   IV   illustrates  the  planner's  role  in  the  application  of 
this  theory  to  the  problem  of  testing  a  guided  missile.     This  chapter 
can  be  read  without  reference  to  preceding  chapters,    if  desired. 


CHAPTER      II 
CONFIDENCE  LIMITS 

Consider  the  problem  of  a  fleet  testing  agency  in  estimating  the 
effectiveness  or  usefulness  of  a  weapon  or  weapon  system.     For  exam- 
ple, a  destroyer  which  is  equipped  with  a  new  or  improved  anti-submar- 
ine weapon.     In  order  to  express  the  effectiveness  of  this  weapon  system 
quantitatively,    some  measure  must  be  used.     Suppose  the  measure  cho- 
sen is  the  percentage  of  hits  achieved  by  the  system  in  a  series  of  inde- 
pendent trials.     This  measure  is  regarded  as  an  approximation  of" the 
percentage  of  hits  which  will  be  achieved  by  the  system,  in  a  future  war 
but  the  actual  percentage  of  hits,    or  the  true  value  of     p     as  it  will  be 
called,   is  an  unknown  quantity.     It  is  assumed  that  this  true  value  can 
be  estimated  by  suitable  testing.     This  estimate  will  be  designated  by 

pe      • 

Assume  that  the  naval  planner  has  a  testing  program  which  simu- 
lates as  far  as  possible  the  combat  conditions  under  which  the  system 
might  be  used.     Also,   that  the  number  of  simulated  attacks  or  trials  will 
be  fairly  large  (50  or  more)  and  will  be  conducted  so  that  they  represent 
a  random  sample  of  observations.     Further  assume,  that  the  number  of 
trials  to  be  made  is  fixed  by  limitations  over  which  the  planner  has  no 
control. 

It  is  intuitively  apparent  that  the  accuracy  of  the  estimate,      p        , 
will  depend  upon  the  number  of  trials  conducted;  the  larger  the  number 
of  trials,      n     ,    the  greater  the  accuracy  of  the  estimate  will  be.      If  we 


designate  the  result  of  a  trial  by     x     ,    then  we  may  consider  that     x     can 
have  only  two  values;     x  =    1     for  success  or  hit,   and     x   =    0     for  a  fail- 
ure or  miss.     The  estimate     p^     may  then  simply  be  the  total  number  of 
hits  divided  by  the  total  number  of  trials.     However,  this  estimate  may 
not  precisely  represent  the  true     p     ,   therefore,  a  measure  of  the  possi- 
ble uncertainty  in     p        is  desirable  since  the  test  result  will  be  used  as 
a  basis  for  making  statements  about  the  true     p     of  the  system.     The 
use  of  confidence  intervals,   or  limits,   provides  this  measure.     This 
technique  is  best  illustrated  by  an  example. 

Suppose    50   trials  of  the  system  have  been  conducted  and  15  hits 

15 
scored;  thus      p      =   -ft?    =    .  30     .     What  statements  can  the  planner  make 

about  the  true  value  of     p      ?    3y  the  statistical  method  known  as  deter- 
mining confidence  intervals,   two  limits,    say      L..      and      L^       ,  can  be 
computed,   he  can  then  say  that  the  true  value  of     p     lies  in  the  interval 
between  these  limits,  but  he  can  make  this  statement  only  with  some 
arbitrary  degree  of  assurance  that  it  is  correct.      This  degree  of  assur- 
ance or  confidence  is  expressed  by  a  confidence  coefficient      a     .     Its 
value  depends  upon  the  degree  of  confidence  the  planner  desires  to  have 

when  he  makes  the  statement  that     p     lies  in  the  interval      L,       to      L-,     . 

i  c 

If  he  wants  to  be  95%  certain  then  the  statement  would  be 

(A)      Prob     (L,    <   p  <   L,)      =      .  95 
where      L,    =   .  17      and      L.?    =    .  43      ,  for  this  example.      This  expres- 
sion  should  be  read:     "The  probability  is    .  95   that  the  variable  limits 
JL,       and      L?      include  the  true  value     p     between  them".     This  implies 
that  there  is  a  5%  chance  of  being  wrong  and  that  the  true  value  of     p 


might  be  outside  this  interval.     A  similar  statement  could  be  made  with, 
say  99%  confidence  (one  chance  in    100    of  being  wrong),  but  if  this  degree 
of  assurance  were  demanded  the  effect  would  be  to  spread  the  limits      JL. 
and      L,?      farther  apart,  that  is,    (.  15  to    .49)   .     Thus,  the  planner  would 
be  more  assured  about  the  truth  of  his   statement  but  at  the  same  time 
less  certain   of  the  value  of     p 

As  Mood  [4]  points  out,    (A)  should  be  carefully  interpreted  because 
it  appears  that     p     is  a  variable  when  actually  it  is  not,     p     being  a  fixed 
value,   the  true  hit  probability  of  the  weapon.     The  variables  are      L., 
and      L.?     .     With  this  fact  in  mind,    (A)    has  the  meaning  that  we  are  95% 
certain  that  the  interval  formed  by      JL.       and      JL-,    includes      p    . 

JL,       and      JL?      are  used  to  represent  the  following  variables: 


P    (1-P    ) 
^e        ce 


Ll       =     .    K  ■-     *-96     |7-5 

jPeU-Pe> 
L,        =  P^     +      1.96 


J2  l*e       ■       *'  /w      \         n 

From   (A)   it  can  be  seen  that  the  limits      JL.      and      L,?      are  functions 
of     p    ,    a     and     n     ,   that  is,   in  functional  notation 

Ll         =         Ll^Pe*    -      n^ 

L2         =         L2^Pe'    -'    n^ 
These  relations  show  that  the  limits  depend  upon  the  outcome  of  the 

test,      p        ,    the  number  of  trials,      n     ,    and  the  confidence  coefficient, 
e 

a     .      Thus,   as  pointed  out  above ,   if      p        =      .3      ,      n     =      50     and     a   =-.95, 

—  e  — 

then      L,,       and      L?      are  determined  as     .  17     and     .  43      ,    respectively. 
The  planner  may  feel  that  the  confidence  interval     .17     :o     .43      , 


computed  on  the  basis  of     p      =    .  3     from  50  trials,   is  too  large  and  that 
if  more  trials  had  been  conducted  he  could  have  located     p     within  nar- 
rower limits.     If  he  had  obtained  :he  same  outcome  of     p      =   .  3     on  tests 
which  had  consisted  of  50,     100  and    1000   trials,    respectively,    the  follow- 
ing Table  illustrates  the  change  in  the  confidence  interval  as     n     and     a 
are  changed: 


No. 

of  trials 

Approx.     95<J 

o  limits 

App 

rox.    99% 

limits 

5G 

.  17      - 

.43 

.  15      - 

.49 

100 

.  21       - 

.40 

.  19      - 

.  43 

1000 

.  27      - 

.  33 

.  265    - 

.  335 

Figure    1    is  a  chart  which  illustrates  the  dependence  of  the  confi- 
dence interval  on     n     and     p       .     For  a  given  value  of     p        ,   a  vertical 
line  intersects  two  curves  corresponding  to  a  given  value  of     n    .     These 
intersections,  projected  on  the  vertical  axis,  are  the  limits      L.,      and 
JL?  ;  and  the  interval  formed  by  these  limits  spans  the  true  value  of     p     . 
Hence,   it  is  labelled  as  the     p     axis.     This  chart  is  for  the  confidence 
coefficient  of     .  95     and  clearly  shows  the  effect  of  increasing     n     . 
Also  the  number  of  trials,      n     ,     may  be  regarded  as  a  function  of  the 
interval      [L«2    -    L.  ]       ,    a     and     p        .      That  is, all  three  quantities 
must  be  known  in  order  to  determine     n     .      In  functional  notation. 

n        =         r-(l-2    "    Ll»    —'     P    ) 


Confidence  Interval  Chart  for 
Confidence  Coefficient  a  =  .  95 

IT:  jure 


It  should  be  noted  that  the  curves  are  not  very  useful  for  attempt- 
ing to  determine,    in  advance,   the  number  of  trials  required  to  give  a 
fixed  confidence  interval  of  desired  length  because  this  would  mean  the 
outcome  of  the  testing  program,      p        ,  would  have  to  be  known  in 
advance. 

It  is  apparent  that  the  use  of  confidence  limits  has  a  place  in  naval 
testing  as  a  means  of  stating  the  results  of  a  testing  program  in  a  pre- 
cise manner  which  is  more  meaningful  than  simply  stating  the  outcome 
of  the  program  as  a  single  number  p  .  But  the  use  of  this  technique 
does  not  provide  adequate  guidance  for  advance  planning  to  indicate  how 


many  trials   should  be  run  or  what  degree  of  confidence  should  be  stipu- 
lated.    This  technique  is  based  on  a  fixed  number  of  trials.      The  planner 
may  have  chosen  the  number,    in  advance,   from  considerations  of  time, 
services  required,    etc.  ,    :u;  we  wish  to  emphasize  that  this  theory  does 
no;  provide  any  criteria  upon  vhich  the  planner  can  case  such  a  ccoice. 


CHAPTER  III 

SEQUENTIAL  ANALYSIS 

Sequential  Analysis  wis  developed  by  A.    Wald  in  1943  for  use  on 
problems  which,  arose  during  World  War  II.     It  was  widely  used  in  manu- 
facturing establishments  for  acceptance  inspection  of  "lots"  of  mass 
production  items.     The  detailed  application  of  sequential  tests  to  such 
problems  is  given  by  the  Statistical  Research  Group  [5],     The  principal 
advantage  of  sequential  tests  in  acceptance  inspection  is  that  it  reduces 
the  amount  of  inspection  required.    As  shown  in  [5],  the  methods  of 
sequential  analysis  can  be  applied  to  experiments.     Since  certain  types 
of  fleet  testing  problems  can  be  thought  of  as  "experiments",   the  appli- 
cation of  sequential  tests  to  a  testing  program  will  be  described. 

Sequential  Analysis  can  be  used  when  a  testing  program  is  to  be  con- 
ducted for  the  purpose  of  comparing  the  hit  probability  or  the  probability 
of  success  of  a  supposedly  improved  weapon  system  with  that  of  the  exist- 
ing system.     Testing  of  this  type  may  be  indicated  when  it  is  desired  to 
use  the  test  results  as  a  basis  for  decision  to  recommend  acceptance  or 
rejection  of  the  modified  system,  for  fleet  use. 

Sequential  analysis  does  not  permit  the  exact  number  of  trials 
required  to  be  determined  in  advance,   however,  an  average  or  expected 
number  may  be  calculated.     From  a  naval  planning  standpoint,   ignorance 
of  the  total  number  of  trials  required  may  pose  some  problems  for  sche- 
duling,  determination  of  material  requirements,   serv.-^s  and  related 
details.     This  maybe  a  disadvantage,  but  the  use  of  sequential  tests  can 
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result  in  ~  possible  economy  of  trials  required  to  reach  a  decision.     This 
economy  may  represent  coniic-er-..^  ^-vings  of  time  and  services. 

In  order  to  use  Sequential  Analysis,   the  planner  must  be  able  to  spe- 
cify certain  quantities  or  inputs.     The  following  hypothetical  example  will 
illustrate  these  inputs  and  how  they  might  be  specified.     The  example  will 
be  similar  to  testing  problems  of  the  type  faced  by  fleet  testing  agencies. 
The  test  results,   which  might  have  been  obtained  for  this  hypothetical 
example  are  shown  in  Table  -. 

EXAMPLE:    A  destroyer  equipped  with  a  supposedly  improved  anti- 
subrr  trine  weapon  is  to  be  tested  to  determine  the  hit  probability  of  this 
system  (designated  as  system  II).      The  hit  probability  of  system  II  is  to 
be  compared  with  the  known  hit  probability  of  otherwise  identical  des- 
troyers employing  a  weapon  which  has  been  in  service  use  (designated  as 
system  I  ).     We  shall  assume  the  hit  probability  of  system  I  to  be     .  2  . 
Also,   we  shall  assume  that  the  cost  of  the  two  systems  is  approximately 
the  same. 

Assume  finally,   that  it  is  desired  to  specify  a  sequential  test.     The 
outcome  of  testing  will  indicate  whether  to  accept  or  reject  system  II  as 
being  better  on  the  basis  of  the  sample  of  trial  runs.     Therefore,   the 
inputs  of  the  Sequential  Test  must  be  carefully  specified.     These  inputs 
uniquely  define  the  seque.-~-^.-  .^ai: 

(a)  p.    -    The  hit  probability  which  would  make  the  new  system  II 

"Unacceptable". 

(b)  p?    -    The  hit  probability  which  would  make  the  new  system  II 

"Acceptable". 
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[c]   c<.    -  The  maximum  allowab  Le  risk  or  probability  of  rejecting  a 

new  system  II  s  ....  probability     p?      or  better. 

The  maximum  allowable  risk  or  probability  of  accepting 

system  II  if  it  has  a  hit  probability  of     p        or  less. 

1 

"  ^  ~"~  ' .r:izy.  Z'S  -  ,  p.  -:  ,  .v  '  In  this  example,  the  first 
ut,  p.  ,  is  probably  the  easiest  to  select.  Since  p,  is  the 
hit  probability  of  a  new  system  which  would  make  it  unacceptable,  it 
seems  logical  that  any  new  system  would  not  be  desirable  if  i':  were  no 
better  than  the  existing  one,  that  is,  if  its  hit  probability  was  no  better 
■jli^-n  :kat  of  system  I,  which  we  have  assumed  to  be  about  .  2.  Hence 
set     p.    =  .  2 

The  second  input,      p?      might  be  selected  by  reasoning  as  follows: 
At  first  thought  it  would  appear  that  any  system  with,  a  hit  probability 
greater  than  that  of  existing  systems  is  "acceptable";  however,    since 
the  test  will  consume  time  and  money  it  would  not  be  logical  to  accept 
a  system  with  a  hit  probability  only  slightly  greater  than  existing  sys- 
tems; say,      p     between      .  2   to      .3      .     On  the  other  hand  it  might  be 
argu^u  that  the  new  system  should  be   at  least  twice  as  good  as  the  old 
zo  justify  expense  of  conversion,   that  is,  an  increase  of     p     to      .4 
wouic  justify  the  expense  of  tests  and  installation  of  the  new  system  if 
it  were  accepted. 

The  inputs      CK.      and      jS      represent  the  probabilities  of  making  a 
wrong  excision  and  these  risks  are  unavoidable,.     The  values  of     C[ 
and     p      are  small  and  no;  necessarily  equal. 

^  is  the  probability  of  rejecting  the  new  system  if  is  has  a  hit 
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probability  o£      .4    .     Since  it  would  be  undesirable  to  reject  a  new  system 
which  is,   on  the  average,   twice  a„  goo< ~  the  existing  one,  then  the  pro- 
bability of  making  such  an  error  should  be  made  very  small.     If  a  risk 
of  one  chance  in  100  can  be  tolerated  then      O^      would  be      .  01    . 

/3     ,   on  the  other  hand,   is  the  probability  of  accepting  a  new  system 
if  it  has  a  hit  probability     p.      .     Since  it  is  possible  for  the  test  to  lead 
us  to  such  a  wrong  decision  it  would  mean  that  we  were  accepting  a  new 
system  which  was  actually  no  better,  perhaps  worse,    in  terms  of  hit 
probability  than  the  existing  o  -~.     Hence,   we  want  to  make  the  probabi- 
lity of  making  such  an  error  .  mall  also.     But,   we  can  tolerate  a  greater 
risk  of  this  error  than  we  can  of  rejecting  a  better  system,    so      jB      can 
be  r;.-uo  Icrger  than      cX.      .      I/,  other  words,   accepting  system  II  when 

it i  the  same  hit  probability  as  the  old  system  is  not  too  serious  from 

a  military  standpoint.     Therefore   if      /S      were  selected  as      .  10    ,   this 
would  be  taking  one  chance  in  ten  of  making  such  a  wrong  decision.     For 
illustrative  purposes,   choose     G\      =    .  01      and       /*>     -   .  10 

Since  either  error  is  possible,   it  would  seem  desirable  to  have  the 
risks,       G\      and       /S      ,    of  making  such  errors  as  small  as  possible. 
That  is,   make      cA      and    p         even  smaller  than  the  values  chosen  above. 
It  will  be  seen  that  demanding  smaller  risks  will  result  in  having  to  make 
a  greater  number  of- trials  and  this  number  can  become  u."^_cceptably 
large;  therefore  some  risk  must  be  tole rated  to  avoid  a  prohibitive  num- 
ber oi  .rials. 

Having  selected  the  inputs 

px   =    .2  p2   =    .4  C*   =    .01  /S  =    .  10 
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it  is  mathematically  possible  to  determine  what  is  defined  as  a  Power 
Curve,      it    (p)    .     This  curve  will  represent  the  probability  of  rejecting 
system  II  as  a  function  of     p      .     It  will  have  the  following  general  shape; 


If(-p) 


;.o 


Pro i       of  reject- 
the  new 

system 


P 

Power  Curve  for  Case  A 
Figure      2 

The  ordinate  at  any  point      p   =   p'    ,   represents  the  probability, 
t;  (p'y     ,   of  rejecting  the  new  system  when  its  hit  probability  is      p' 

p   =   p2   =    .  4      ,      it  (p)      has  the  value  of       CX.  =   .  01      ,  *which  is  the 
probability  of  rejecting  system  II  when  its  hit  probability  is      .4      .At 
this  point  we  are  taking  one  chance  in  100  of  rejecting  system  II  when  its 
true      p      is  equal  to      .4      .     Xote  that  if  system  II  has  a      p    >     .4    ,    we 
have  a  still  smaller  probability  of  rejecting  it. 

At  the  point      p   =   p.    =    .  2      we  have  a  very  high  probability  of  reject- 
ing the  new  system  and  since  (one  minus  the  probability  of  rejection)  is 
equal  to  the  probability  of  acceptance,    then      [1       -      -rr  (?-,)]=      ft     =.10 


:«* 


This  is  the  risk  we  are  willing  to  take  in  accepting  the  new  system  when 
it  is  no  better  than  the  existing  one. 

Between      p.       and      p.,      the  new  system  will  be  rejected  with  probi- 
lities  varying  from      1    -   )S     to   OL  -  the  probability  of  rejection 

decreasing  cs  we  approach  the  "acceptable"  hit  probability  of      .4 

The  Power  Curve  need  not  actually  be  produced  in  order  to  make 
use  of  sequential  testing. 

USE    OT   THE  T2ST       The  four  inputs,       p.     ,    p_    ,     <*.      ,    ft     are 
used  to  construct  a  graph  which  is  the  basis  of  the  sequential  test: 


.-. 


Graph  for  Sequential  Test 
Figure      3 


The  graph  is  easy  to  use.     Assume  that  the  results  given  in  Table  1  are 
bein^j  ob  —  ined  from  a  testing  program  where  the  outcome  of  each  trial 
is  denoted  as  success  or  failure.       After  each  trial,  the  total  number  of 
successes  is  plotted  against  the  total  number  of  trials  conducted  thus 
far.     As  trials  progress  the  plotted  point  will  either  fall  in  one  of  the 
shaded  regions  labelled  Acce;c  ;  System  II  and  Reject  System  II;  or  it  will 
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be  in  between  the  parallel  .lines.      Trials  are  continued  as  long  as  the  point 
remains  between  the  parallel  lines  but  eventually  one  of  the  shaded  regions 
will  be  reached.     It  is  this  uncertainty  as  to- when  one  of  these  regions  will 
be  reached  that  precludes  advance  determination  of  the  number  of  trials 
reqv 

e  ^~™iis  of  constructing  the  straight  lines  which  comprise  the 
'^ou:^...-ies  of  the  three  regions  is  given  in  Appendix  A  . 

j.^'-jis    6,    7   and    8  are  constructed  using  z'ae  same  te'st  results 
^iven  in  Table  1,  for  three  different  sets  of  input  parameters,   and  will  be 
described  as  case    (A),    (3)   AND    (C). 

Trial  number    i  2  3  4  5        6  7  -9  1011 

Outcome  FFSFSSFFSSF 

12         13         14         15         16      17         18         19      20         21         22 
FFFSFFSSFFF 

23    24    25    26    27   28    29    30   31    32    33 

F    F    F    F   •  F   F    S     SS     S     S 


Table  of  Assumed  Test  Results 
Table      1 


CASS      A 

_"igure    6    illustrates  the  results  that  would  have  been  ob^ined  u^mg 
the  values 

px    =    .2  p2  =    .4  cX  =   .  01  p    =    .  10 
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Note  that  acceptance  of  System  II  as  having  a  hit  probability     .4     or 
greater,   would  have  occurred  at  the   19th  trial  where  the  plotted  point 
crossed  the  boundary  into  the  acceptance  region. 

The  Power  Curve  for  this  particular  set  of  input  parameters  has 
been  shown  earlier. 

CASE      3 


o 


parameters: 

px  =   .2  p2  =    .4  c*    =   .01  P    -   .05 

Here,  the  power  curve  will  appear  much  the  same  as  for  CASS 


(A)   but  note  that      p>       is  now  smaller. 


/ 


Prob.    of  reject- 
ing new  system 


.  2  «4- 

Power  Curve  for  Case   B 

Figure      4 


..o 


"'? 


Decreasing      B      from   .10    in   Case(A)    to  the  value  of     .  05     in  thi; 

case,   is  equivalent  to  saying  that  the  planner  wants  a  smaller  risk  of 
accepting  System  II  if  1:  :^  no  better  then  System   I  .     A  decrease  in  the 
risk  might  be  considered  necessary  because  of  the  economic  aspects, 
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such  as  the  desire  not  to  take  too  great  a  risk  of  char   ;ingto  a  new  sys- 
tem which  is  in  reality  no  better  than  the  existing  one,   if  the  change 
represents  considerable  expense.     This  is  equiv;  lent  to  desiring  more 
protection  against  the  risk  of  ;    wrong  ^^cision  and  will  result  in  more 
trials  being  required. 

Figure  7  -hows  the  test  data  plotted.     The  boun. ries  -i-e  different 

from  C ^A),   as  indicated  by  the  equations  of  the  lines. 

_oy   '.he  acceptance  o£  System  II  results  at  the  33rd  trial  as  opposed 
to  the  19th  trial  in  Case  (A).     This  increased  number  of  trials  is  the 
result  that  was  anticipated.     The  number  of  trials,   for  the  smaller  risk 
/3      ,   is  greater. 

Input  parameters: 

?i   =    .2  p2  =   .6  <A  =   .01       7      /3    =   .  10 

Before  ---cussing  test  results  in  this  case,   it  is  important  to  note  the 
imp'.-c ons  of  setting      p.    =   .  6 

The  power  curve  will  have  the  following  general  shape 


Prob.  of 
rejecting 
new  system 


Power  Curve  for  Case  C 
Figure      5 


oC  =  .01 


i.o       I 


1- 


and      C\     are  ;ame  as  in  Case  (A)  but  -----  planner  in  now 

say  2  will  accept  the  new  weapon  on  the  assumption      p~    =    .  6      and 

ke  the  same  risks        G{  =    .  01      of  rejecting  the  new  system  for 
p    -    .  6      .     Note  from  the  pow^r  curve  that  if  the  true      p      of  System   II 
is  in  the  neighborhood  of  .4   or    .  5   he  is  —king  a  greater  chance  of  reject- 
ing  System   II  than  in  Case   (A)  .     This  is  just  another  way  of  saying  that 

he  is  not  too  concerned  with  the  interval  between     .  2     and     .  6     so 

;_si  will  reject  systems  which  have  a  true  hit  probability  in  this  region 
th  a  higher  probability  -nan  in  Ca^o  (A)  .     Or,   System   II  must  have1  a 
higher  hit  probability,   on  the  average,    than  in  Case  (A),    to  be  accepted 
by  this  test. 

Figure  S  shows  the  result  of  the  test  data.     The  boundaries  are    " 
again  a— Cerent  than  those  of  Case  (A)  and  (3)  „ 

Note,   that  on  the  28th  trial  the  sequential  test  indicated  re'jection  of 
_    _..    -_  as  not  having  a  hit  ^.robabi'nky  of     .6     or  grcw^or. 

It  is  important  to  recall  that  in  any  of  these  three  cases  there  is 
always  the  chance  that  a  wrong  aecision  will  be  made.     We  have  tried 
to  keep  the  chance  or  risk  of  such  eventualities  small,   namely  by  keep- 
ing      C\     and       /S      small,    consistent  with  a  reasonable  number  of  trials. 
A  demand  for  less  risk  of  wrong  decisions  will  result  in  more  "trials '' 
being  required.     Further,         o\     and        /3      had  to  be  selected  on  what 
might  be  called  an  "intuitively  reasonable"  basis.     The  sequential  test 
does  not  provide  any  means  whereby  the  cost  of  testing  and  cost  of 
wrong  d^c.nor.  can  be  reconciled. 
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Lg. 

i   A,    B    e C    ,   it  should  be  noted  

ter —  acceptance  or  rejection  ~re  used  with  reference  to  assumptions 

wses  about  the  new  system.     In  other  words,   a  st cal 

the  :e~..  of  an  assumption.     In  c .   A  and  3   we  tested  the 

asis  that  System   I-  had  i  robability  of     .4     and  as  a  result 

of  the  t( we  accepted  this  h    -othesis.     In  Case   C    ,  rejection  of  the 

System    II  was  indicated,, 

i  be  reasons  for  dec. g  not  to  use  a  new  system  e . 

if  the  s stical  results  in< _ — d  that  it  was  desirable.     For  example, 

2  new  system  maybe  too  heavy,   cr  useless  in  rough  seas,   or  require 
excessive  maintenance.     If  many  such  factors  weighed  again.,.,  a  new- 
system,   obviously  it  would  not  be  ceceptable. 

2„       The  example  given  is  just  one  illustration  ci  the  use  o_*  sequen- 
tial tests.      They  are  not  lim e.  to  trials  ;o  determine  hit  probability 

[Binom  ^tributions)  but  could  also  be  used  in  trials  where,   for 

Le,    the  miss  dis:„/.eo  between  weapon  and  target  is  being  mea- 
sured.    While  the  details  of  the  example  used  in  this  chapter  would  not 
apply  to  such  a  test,   the  p  as  are  the  same.     See    [5]   . 

It  has  been  mentioned  that  advance  planning  maybe  handicap- 
ped  because  the  exact  number   -1  trials  cannot  be  specified  but  this  fact 

should  not  preclude  the  use  of  Sequent: I  Analysis.     In  many  situations, 

the  knowledge  of  the  expected  numbei  s  required  may  be  suffi- 

cient for  planning.      The  great  improvement  e_"  Sequential  Analysis  over 
the  classical  test  procedures,    namely,    the  reduction  in  the  number  of 
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-----  -  e. iecisio 

.and       /S      are  inputs  to  the  probl  hich  a 

economic  anc  ary  considerations.     The  concept  of  cost 

Lied  when  a  value  of  '    .t  pro  bability  of  an  "acceptable"  sys 

osen.     Also,  -  the  corresponding  risk       OC    of  reject- 

_  sg    si 1  a  system  is   specified,    there  is  an  associated  idea  of  cost. 

Further,    there  is   the  cost  of  testing  program  which  might  include  ser- 
vices,   material,    cost  c_'  the  exj  ended  in  teo.,    etc.     These 

cos:-,  are  mentioned  to  point  out  th although  the  planner  consid< 

them      hen  specifying  the  inputs,    the  theory  docs  not  permit  the  planner 
to  e  take  such  cc^~~  into  account. 

-.  moro  gcnerul  theory  scribed  in  me  ncx':  chapter.     Explicit 

sions  for  :o„:  of  tes g  and  costs  of  wrong  decisions  provide  a 

means  for  determining,   prior  to  testing,   the  number  of  trials  to  be 
conducted. 
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CrL              .     IV 
riCALDI         HZ  1  CLV- 
ne  ral  theory  al  planners  at  the  CK 
/•el.     In  order  to  discuss  the     pplic  .tion  of  the  theory  it  seems  appro- 
scribe  a  situation  g  i   ing ;e  to  a  Statistical  Decision 

jblem. 
_____ 

1.       leed  for  an  Air-to--A  -ded  Missile  is  recognized  and 

tran„__^  to  an  Operational  Requii  t  by  an  office  of  C1\~C 

2.  The  Operational  Requirement    -isses  to  t  sant  11-ureau. 

A  con' ct  is  let  and  a  missile  is  produced.     At  this  point  testing  by 

Bur. ar.d  contractor  is  invol    3d,   but  this  pape:    is  not  concerned  with 

ise its.     At  some  future  -----  the  Bureau  will  inform  CXG  that  a 

missile  has  been  produced  to  satisfy  the  Operational  Requirement. 

3.  It  is  assumed  that  CNO  now  -3  an  estimate,      p        ,    of  the 

1  e 

perce ge  effectiveness,    or  probability  of  success,    of  the  new  missile 

-     2  combat.      In  order  to  obtain  this  estimate,   missile  testing  by 

some  et  agency,    such  as  OpDevFor,    is  radicated.     An  office  of  CNO 

must  ------  how  many  weapons   should  be  tested  (that  is,    how  many 

trials  should  be  conducted)  so  that  the  testing  agency  may  plan  the  test- 
program.     This  question  of  fcl  aber  of  trials  required  consti- 

tutes  the  Statistical  Decision  P: >lem. 

; tistical  -.-sision  Theory  provides  a  rational  basis  for  answer- 
ing such  questions,    moreover,   the  answer  is  provided  prior  to  testing, 

25 


is  from  v.ew  of  planning.     In  chap- 

ter  \  -  /er  provic^c.  by  this  ...^ory  in 

a  sp^c. set  of  i ■< nstances   which  will  be  related  to  ;h^  problem  posed 

re  example.      This  special  set  cumstances  includes: 

(a)  In  — s  trials  from  which  the  e late     p       is  obtained,   the  c_> 

e 

come  of  y  trial  must  not  be  ehiected  or  influenced  by  the  out- 
come of  preceding  tr: s,    -„  e.  ,    t  -Is  must  be  statist! 

independent. 

[b]  Probability  of  success  of  the  missile  is  defined  as  the  probabi- 
lity of  successful  laui  Ling  and  detonation  of  the  mis- 
sile at  intended  point.     In  actual  combat  there  will  exist 

some  value  for  this  probability  ci  .uccess,   which  we  will  call 
the  true  value  of     p     .     This  value  is  an  unknown  quantity  but 
can  be  estimated  by  tes  ting. 

rJlie  outcome  of  each  ti  al  is  value< s  one  for  success  and 

zero  for  failure.     Note     ..at  _h._l__^  here  :.o  ;.o':  concerned  with 

wh: of  the  three  phases  of  (b)  fails.     In  any  series  of  inde- 

pendent  trials,   an  estimate  of  the  true  probability  of  suc- 

e 

-„_s     p     is  given  by  the  number  of  successes  divided  by  the 

- . -  :/-;ho e  r  of  tr  ial s . 

en  these  special  circumstances  are  fulfilled,   then  an  answer  is 

given  _ _  -hrm  ox  'tables  presented  at  the  end  of  the  chapter.     The  use 

of  these  tables  requires  a  specification  of  certain  inputs  to  the  problem. 
These  inputs  will  be  explained  in  the  light  of  the  hypothetical  testing 
>blem. 
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1 

at  represents  the  concept  ;hat  the  true  value  of     p     ,    since 
it  is  a  pr<  I  /ty,   can  only  take  on  values  between  zero  and  one.     Hence 

it  can  be  visualized  as  any  point  in  the  interval      [0,    1]    .     In  this  sxl.  .- 

of     j     in  this  range  c--~  ec±- lly  likely. 

sly  says  that  the  planner  has  no  a  pri  :nowled  ;e  ing 

of  thv,  thers.     In  technical  la ngi — ^e,   he 

. ;   o^wCiiijc  iio ;;:.  probability  density  function,       f.    [p)      ,  for     p  „ 

Lcally: 


l.o- 


-?>  -p 


:.o 
Uniforn  -  ens ity  Function 


er  I ids    if  the  planner  had  some  a  priori  knowledge 

whic  -      lead  him  to  xvor  certain  values  of     p     ,   he  might  specify 

_  proba' ty  density  function  \    -_cn  reflects  this  knowledge,    ^nch  as: 


*  f 

o  i.o  ' 

Triangular  P:  .ility  Density  Function 

Fig*  .3      10 
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in  the  ru  rhood  - 

..ers, 
ner  specifies  an  a  t     p     as  we 

Le   solution  we  wi]  3  the  Statistical  Decision  Problem 

If  an  a  prio  ._  ier function  is  not  sp^- 

2  solution,   if  any,   is  terme  .snax  solution.     '..  Linim 

solu  not  be  considered  i  -  thesis.     An  elem^.  scus- 

alutions  is  given  by  Tucker  [l],  .-.-. _  _-  more  advance 
atmei  d  [2]  . 

-ut  ■■' .   .   .."inal  cr  Terrr-i'.'-.",.  Dec:;,is^s 

te  estimate,      p        ,   will  be  the  basis  for  making  decisions  con- 
cerning the  missile.     Hence(   it  is  necessary  to  s~pcc.-  >n  is 

an  after  a  value  of    p       ha~   -  een  determined.     Action  will  result 
from  ./e  shall  call  termii  Lons. 

In  order  to  apply  the  theory,    the  planner  must  be  _-/.e  to  specify  a 
mber    vhich  we  shall  refer  cc  as      i     in  the  int  3,    __      .     This 

value  of     8     is  used  to  specify  two  te  ...  decisions, 

d      >     0      will  lead  to  a      sci:  ion  to  ad              ;  missile  and  v. 
imply  that  flee':  units   will  be  converted  to  the  \ ie  oi  mis- 

sile.     We  shall  designate  this  first  terminal  excision  as  the 

t 


eels  ion  to  Convert  o£  briefly      d. 


(2)     p^    <      8      will  lead  to  the  decision  to  order  fu    ;he]    develop- 
ment of  the  missile.      This  decision  would  _.v.  Lat  the  mis- 
h  ls  shown  promise  of  being  feasible  and  there  is  a  reason- 
pectation  of  imp                missile  performance  to  a  point 

1    ~, 

Chapter   3 


...  warrant  its  au:_ n  for  fleet  use.     We  shall  desig- 

nate  this  second  terminal  decision  as  the  Dect'oior.  to  Develop^ 
or      d2       . 
The  value  -ssigned  to     0     mig  -ed  by  a  comparison  of  ::.^ 

ass  of  existing  2  A/C   weapons  and  determining  what 

val\_v-  of  probability  of  success,    if  attainable  by  the  new  missile,  would 
rep:---^'..  ^;  l^a^'j  satisf  /  or  acceptable  interceptor  effectiveness. 

Suppose     8     were  chos^/.  as      .0     .     This  would  be  equivalent  to 
stating  that  a  missile  with  _  probability  of  success  of     .  5     will  result 
sceptable  or  satisfactory  raterceptor  effectiveness  against  enemy 
bomber  capabilities  for  the  immediate  future.     Further,   it  should  be 
a  value  which  ii              ~    ,.ble  to  expect  in  view  of  missile  performance 
during  : — ~  cou;.e  of  develop.'.. aat  ava.,  far. 
'.  :-f  '."•■■:•.•■■  Decisio 
Kaving  defined  the  terminal  elisions  it  is  now  important  to  note 
that  :he  test  result,      p        ,    can  lead  to  a  wrong  decision  in  two  ways. 
To  understand  this,   the  difference  between     p       and     p     must  be  kept 
in  mind.     We  would  like  to  tl        : t     p       was  always  an  "accurate" 

ss ' e  of      >     ,    but  such  is  not  the  case.      The     p       indicted  by  the 

"  e 

tests   o^-n  be  greater  or  less  than  the  true  value  of     p     by  an  unknown 
amount.     The  two  cases  which  lead  to  wrong  decisions  are  shown 
graphically: 

(  See  following  page  for  figure  ) 


2S 


-< 


e 

(a) 


T> 


e 
(b) 


..- 


Ca^_  _  .-       rong  3.c-„o.-.o 

Figure 

Figure    11    (a     .3  the  cas^  whe    -  the it  result,      p        ,   is  greater  than 

e  ^ 

while j   in  fact,   the  true  value  c_'     p     is  less  than     8    .     In  such  a 

case  the  Decision  to  Convert  would  be  made.     It  would  be  a  wrong  ^..o 

sion.     In     b      ,    the  test  result,      o        .    -  '.ess  than     -0     while,   in  fact, 

e 

the   -     ...-ue  of     _^     is  greater  than     6     .     In  such  a  case  the  Decision 

to  Develop  would  be  made  as..:,  it  would  be  a  wrong  decision. 

Inp    t      III      Tl      Cost  of  Wrc      •  !3s;  is      (Weight  Function) 

It.  is   through  the  Weight  Function  that  cost  „-'  wrong  decision  is 

introduced  into  the  problem.     'ore  specifying   the  form  cJ  :s..3  func- 

tion  it  is  necessary  to  understand  the  nature  of  these  "costs"  in  this 
imple.  First,  they  do  not  represent  cost  in  the  sense  that  money 
ist  be  i — _  out.      They  represent  the  "value"  of  the  possible  conse- 

quences  suffered  as  a  result  of  a  wrong  excision.     Refle< on  will  ^-ow 

-  this  definition  can  lead  to  situations  which  cannot  be  m< sured  in 

terms  of  dollars.     For  instance,  je  as  a  result  of  a  test  the  mis- 

sile is   "accepted",   i.e.    Decision  to  Convert  is  ^use.     Suppose  a  war 

er s  and  we  find  our  interceptor  \.  p  srformance  almost  nil    1 

is,    the  true     p     is  very  low)  .     This  could  lead  to  loss  of  a  Task  Fa 
due   to  lack  of  protection  -  the  Task  Fc.  :        >ss  could  mean  delay  o: 


30 


inab  •   sxecute  the  c  Li      Lon.     Obviously,   it  is 

imp  .ssign  a  dol ■  o  such  consequ...:^..     If  this  could 

be  d  be  taken  into  ace  =  >ry. 

.her  than  attempt  to  e.  evaluation  of  losses  due  :o  wrong 

decisions  to  such  length,    i.  is  consic  sred  reason—--.,  , — -  ^---e~-c:.-  to 

.  of  the  ••-c_.,;  of  a  ecisi s  an  c-rc"..  :...      . :•--      of 

doing  the  planner  will   ae   ev<         a  Lg  the  losses  or  cost 
-_^ions  in  terms  of  the  dc  :  jet  under  which  he  must 
operate.      These  losses  can  no\*  be  associated  with  consequent  cost  of 
ier  of  the  two  terminal  decisions. 

1.  When  the  Decision  to  Convert  i-  made,    the  cost  of  this  deci- 
sion may  include: 

[a)     Cost  of  convei  . ..      fleet  aircraft,    end  fleet  units  :o  use  the 

missile,   that  is,    cost  of  structual  changes,    necessary 

"e  control  systems,   etc. 

ting  i • !t  may  not  be  suitable  and  new  aircraft, 

designed  around  this  missile  and  its  fire  control  system, 

may  be  need--, 
[c]     Cost  of  actual  missiles  required  by  the  fleet  for  some  sub- 

sequent  budgetary  periods    that  is,   mass  production  costs. 
Let  us  -ssume  these  costs  to  be  on  the  order  of  $10,000,000. 

2.  eon  the  Decision  to  Develop  is  made,    the  cost  of  this  decision 
will  include : 

(a)     Cost  of  additional  development  work  by  contractor  and 
services  incident  to  the  word  to  develop  the  missile  fur- 
ther.     This  will  represent  some  fraction  of  the  total  devel- 
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...  funds  ava        -le. 
ssume  sost  is  also  on  the  order  c-  ... .  J^  000,  000. 

tot  require  the  costs  o£  (1)  and    (2)   above  to  be  equal 

.  the  so! on  is  simplified  if  the]  il. 

o  information  o  rids  to  cover  such  cases  as 

lis  .^^  above  are  identifiable  in  the  mil: ry  budg< It  is  assumed  that 

there - gories  of  funds  :h        e  sither  ear-marked  for  fleet 

improvements  cf  the  nature  of  those  in    (1)    ,   c  elopment  workas 

contemj d  by   (2)  .     The  me point  is  that  Lite  costs  must  be 

associatec  t  these  decisic.-..    i  nd  the  planner  must  be  able  to  provide 

r sonable  estimates  cf  these   - 

f  :!'-  ^ 


The  form  of  the  Weight  F            .on  describe,  the  manner  in  which  the 
planner   will  penalize  or  asses.  xa when  a  wrong  decision  is  made. 

Refer  .g  Figure    11   (a)   .      A   -en  the  Decision  to  Convert  is  a  wron 

a^eisioy.,    we  would  be  spending  conve-^.e a  faaas  for  _  missi  zh  

not  meet  our  criterion      (9  =  .  5]      for  acceptable  performance.      One    .  ay 
tc  _.  -  -  .elves  for  this  v,  i  .  n.g  decision  would  be  to  make  the  p 

—  y  proportional  to  the  amount  '        [   ;he lef Is  to  .meet  the  c  rite 

For  instance: 

True  value  of     p 
.5 


enalty 
0 

4C- 
60%  " 
80%  "   . 
100%  " 

conversion  cost 
n                  ii 

it                  ii 

u                   ii 

ii                  ii 

32 


loo  - 


iver Cc  st 


Z     -5    A 


i.o> 


■  - 

Penalty  Assoc  —  ed  with     d 


12 


-? 


This  graph  shows    .        t  we  can  call  the  cost  of  wrong  decision  associated 

d.        .      It  is  defin< 3  a  Weight  Function,  Symbolically: 

.    ...    ]      .     It ic that  we  penal o elves  nothing  if  the 

true      p    >  .  5  since  a  missile  wi-h  1  >     i'a  ' range  is  as 

setter  than  ..  e  asked  f<  r.     At       p  =    .4      we  have  a  missile  almost 
as  good  so  the  penalty  is  nc  very  great.     Similarly,    the   smaller  the 

of     p      j   the  greater  the  penal/ty  for  spending   :on  'ersion  funds. 

similar  reasoning  we  can  define  a  cost  for  the  other  wrong  deci- 
sion  ..—  ch  can  be  made.     Refei  to  Figure    11    (b)  .     When  the  Decision 
to  Develop  is  a  wrong  decision,   money  is  spent  to  develop  a  missile  which 
is  already  as  good  or  better  th  n  the  criterion     (6  -  .  5}    „     We  coa'.w 
therefore  penalize  outselves  ir  proportion  to  the  amount  that     p 
the  criterion. 


-3  oi 

m  4 

.2 

.  1 
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de 

nent  cost 

11 

it                     ii 
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M                     ii 
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ii                     it 
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Perec  of 

Development  Cost 
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Penalty  Assoc:  .     d 


_■  lg  are 

s  is  the  graph  of  the  cost  of  the  s  ,  decision 1  can  bo 

made  c  "       ..'    -   ..  [p,   d-  )    „     There  is  no  p<  fc  -      p  <   .  5      because 

lissile  with  a  value  of     p     in  that  range  requi: fi — - development, 

according  to  our  criterion.     But  for  a    falue  of     p     ,    say      p  =   .  75      , 

-  e  penalty  should  be  assign  ;d  s a  missile  with  this  value  of 

does  not  require  further  nent.      Similarily,   as     p     increase 

penalty  increases.     By  this  process,  the  greate  ralue  of     p     ,    .... 

greater  the  penalty  for  spending  development  funds. 

If  the  increments  in  Fig' 12  and   13   are  imag to  get  sms 

and  _.     •  the  discrete  fi ion  will  approach  a  straight  linec     This  is 

equivalent  to  defining  a  linear  Weight  Function,,     To  sii  ;he  mathe- 
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7 
.--  ..  :al  details  the  costs  of  conversion  and  development      w-^    )      are 


assig  led  the  -  of  one  value  unit. 


vtfatf) 


W(f>jcl£) 


Value 
units 


^ -■•-'" 


•P 


where  Dec.„c:. 
.  )  Convert  is  made 


6-"^' 


Case  where  D< ion 

■;o  Develop  is  made 


'. — lear  Weight  Function 
-".jure       14 

Consider  what  Figure    14    (a)   indicates.      There  is  some  penalty  for 
every  value  of     p  <  6      .     Suppose     p     is  in  the  interval     .4     to     .5     . 
A  missile  with     p     in  this  range  is  almost  as  effective  as  one  with  a     p 
of     .5     .     Consequently;   when  the  decision  to  convert  is  made,   it  seems 
logical  that  there  should  be  no  penalty  in  ~ases  where  values  of     p     are 

onl}    ghtly  less  than     8      .     Prec:-„.yy   we  can  define  an  interval 

[9,    8]       where      W(p,  d-    )      =      0    . 

For  -"-jure    14    [b]    ,    similar  reasoning  will  apply,,     Suppose     p     is 
in  the  interval     .5     to     .6     .     A  missile  with     p     in  this  range  requires 
only  slightly  less  further  development  than  a  missile  with     p     of     .  5    . 
So,   for  values  of     p     slightly  0reater  than     8     ,    it  is  also  logical  that 
there  should  be  no  penalty  for  aie  excision.     Or,   define  an  interval 
[6,    1-8]      where      W(p,   d^)   =    0   . 
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The  Weight  Function  can  now  be  graphed  as; 


Wipjdf) 


Wfec/i*) 


Value 
units 


6  d-d) 


TT*P 


(a)   Case  where  Decision 
to  Convert  is  made 


(b)   Case  where  Decision 
to  Develop  is  made 


Linear  Weight  Function 
Showing  the  Parameter     8 

Figure       15 


The  value  of     8     is  at  the  discretion  of  the  planner.     For  this 
example,       8      is  chosen  as     .33 

Another  possible  form  of  Weight  Function 

It  should  be  noted  that  the  linear  form  of  the  Weight  Function 
resulted  from  the  way  in  which  it  was  assumed  that  the  cost  of  wrong 
decisions  could  be  assigned.       As  Wald  [2]  points  out,   the  Weight  Func- 
tion is  the  most  difficult  input  of  the  Statistical  Decision  Problem  to  deter 
mine.     He  further  points  out  that  a  Simple  Weight  Function  is   suitable  for 
many    practical  problems. 

A  simple  Weight  Function  has  only  two  values,   zero  or  one. 
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Value 
units 


*(f'dl) 


1  - 


O  8    0  lo 

(a)   Case  where  Decision 
to  Convert  is  made 


^rr 


-+— ^ 


e  (/-a)       i-o 

(b)    Case  where  Decision 
to  Develop  is  made 


r 


Simple  Weight  Function 
Figure       16 

The  meaning  of      6    ,    p    ,     3    ,     d  and      d?        are  exactly  the  same  as 

before. 

The  difference  between  this  Simple  form  and  the  Linear  form  is 
that  the  cost  of  wrong  decision  is  not  proportional  to  the  difference 
between     p     and     0     .      There  is  no  penalty  to  be  assessed  for  a  missile 
whose  true  probability  of  success  lies  between     8     and      (1-8)      .      But, 
when  the  decision  to  convert  is  made  and     p  <    8    ,    the  penalty  is  the 
entire  cost  of  conversion.     Similarly,   when  the  decision  to  develop  is 
made  and     p    >      (1-8)      the  penalty  is  the  entire  cost  of  further  devel- 
opment. 

The  Simple  Weight  Function  has  been  illustrated  so  that  compari- 
sons of  solutions  using  Simple  and  Linear  Weight  Functions  can  be  made 
Input      V  Cost  of  testing  of  Cost  Function.        (C) 

This  input  is  not  as  difficult  to  specify.      In  many  testing  programj 


37 


the  cost  of  each  trial  in  the  program  is  the  same.     When  this  is  the  case, 
then  the  cost  of  testing  is  proportional  to  the  number  of  trials. 

As  in  the  other  costs,    the  naval  planner  must  be  able  to  estimate 
this  cost  of  testing.      This  would  include  such  items  as  the  cost  of  the 
missile  itself;  contractors   services   required  for  the  tests,    etc.         C     is 
therefore  determined  by  a  cost  analysis  of  charges  directly  applicable 
to  the  testing  program  for  a  specific  missile.      It  is  not  considered  that 
the  cost  of  operating  the  testing  agency  for  the  duration  of  the  testing 
program  is  a  cost  which  would  be  included  here. 

Suppose  the  cost  of  each  trial  for  the  missile  in  this  example  is  on 
the  order  of  $50,000.     As  it  is  used  in  Statistical  Decision  Theory,       C 

is  expressed  in  the  same  value  units  as  the  Weight  Function.     For  the 

7 

Weight  Function  with  the  maximum  value  of  $10      =1    value  unit,    then 

define      c    =    C/W    =    .  005      value  units  . 

The  effect  of  varying  the  ratio   C/W    will  be  discussed  in  the  next 
chapter. 

(B)    Solution  of  The  Example 

The  preceding  discussion  has  indicated  how  the  inputs  required  for 
a  Bayes  Solution  of  a  Statistical  Decision  Problem  may  be  specified. 

The  solutions  are  given  in  the  form  of  Table  2,    using  the  Linear 
Weight  Function  and  Table   3,    using  the  Simple  Weight  Function. 

By  way  of  summary  of  the  problem  to  which  the  Tables  are  solu- 
tions,   we   shall  briefly  review  the  special  circumstances   required  of  the 
testing  program  and  the  four  inputs  to  the  problem. 

(a)        Recall  from  Page  26  that  a  suitable  testing  program  will  con- 
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sist  of  independent  trials,   the  outcome  of  whi        is  either   suc- 
cess or  failure.      The  estimate     p        is  the  nu;       or  of  successes 
divided  by  the  total  number  of  trials. 

(b)  Input  I  .  The  specification  that  the  true  p  (  he  missile  can 
take  on  any  value  in  the  interval  [0,  l]  anc  at  all  values  of 
p     are  equally  likely. 

(c)  Input   II  .      The  final  or  terminal  decisions  anc        eir  relation 
to  the  preassigned  value  of     0    =    .5 

d.      -    the  Decision  to  Convert 
d?      -    the  Decision  to  Develop 

(d)  Input   III    .      The  cost  of  wrong  decision  or  We  Function   (W); 
both  Linear  and  Simple  forms. 


.         .   t)    -    Cost  or  penalty  when      d,        is  a 
(P.   dj  r  1 


clq  decision 


W 

"         Lt  t 

W(p,    d?  )    -    Cost  or  penalty  when      d?        is  a  •         ag  decision 

The  maximum  value  of   W    is  assumed  to  be     !|  (1    value 

unit)  and      d    =    .  3  3 
(e)     Input   IV    .     The  cost  of  testing  or  Cost  Functi         (C) 

Testing  was  assumed  to  cost  $50,  000  per  tria         nee, 

c    =   $50,000/$107    =    .005      . 

Note  that      c      has  the  same  value  as  long  as  t.         atio    C /W    is 

the  same,    i.  e.  ,    if  we  had  assumed   W    to  be  $  and  Cost  per 

trial  of  $5,000  then      c      is  again     .005     . 
Explanation  of  Table  2  (Linear    W)  ;  page    40 

Any  cell  in  the  table  can  be  identified  by  a  pair  of  i         bers  or  coor- 
dinates,   say     (i,   j)      ,   where     i     denotes  the  number  of        lures  and     j 
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NUMBER  OF  SUCCESSES 


0 

■ 

0 

F 

F 
A 
T 

L 
U 
R 
E 
S 


0 

1 

2 

5 

4 

0 

i"  I6667" 

;. 02466 

.03704" 
.01966 

"1 .00926 
|,00926 

.0021*7 

.0024? 

.00069 
.00059 

1 

1.03701* 
'.01966 

.09259 
.02546 
-j 

.02963 
.02046 

j . OO96O 
-.00960 

. 00514 
.00314 

2 

.00926 
.00926 

1,02963 
;. 02046 

.05967 
.02426 

.0225UI 
.01920! 

1 

,00340 
. 00840 

3 

.00247 

.00247 

.00960 
.00960 

1,02254 

'.01926 
1 

,04133 
. 02206 

,01724 1 

,01706; 

k 

.00069 
.00069 

.00314 
.00314 

,00840  1,01724 
,00840  1.01706 

,03000  I 
. 01335 j 

Solution  Using   Linear  Weight  Function 


Table 
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the  number  of  successes  which  have  been  observed  in     (i   +  j)  trials. 
The  numbers     i     and     j     are  the  numbers  designating  each  row  and  col- 
umn respectively.      Thus,    row  two  and  column  three  locate  a  specific 
cell  after  trials  which  yield  two  failures  and  three   successes. 

Every  cell  contains  an  upper  and  lower  entry.     Each  entry  repre- 
sents a  cost  expressed  in  value  units.     If  desired,    these  entries  may  be 

converted  to  dollars  by  multiplying  by  the  dollar  equivalent  of  a  value 

7 
unit    ($10      in  this  example)    .      The  upper  entry  is  the  expected  cost  (in 

value  units)  if  no  further  trials  are  made  and  a  Terminal  Decision  to 

Convert  or  Develop  is  made  on  the  basis  of  trials  thus  far.      The  lower 

entry  is  the  expected  cost  (in  value  units)  if  trials  are  continued  and  a 

Terminal  Decision  to  Convert  or  Develop  is  based  upon  the   result  of 

further  trials. 

The.  relation  between  the  upper  and  lower  entries  means  that,   in 
any  cell  where  the  lower  entry  is   smaller  than  the  upper,    the  expected 
cost  is  less  to  continue  testing  than  it  is  to  make  a  decision  at  this  point. 
Notice  that  the  cells  in  which  the  lower  entry  is   smaller  than  the  upper 
are  enclosed  by  the  dotted  line. 

It  is  this  dotted  boundary  which  enables  the  planner  to  determine, 

in  advance  of  testing,   the  limits  on  the  number  of  trials  required.     The 

Table  actually  yields  the  minimum  as  well  as  the  maximum  number  of 

trials  that  might  be  required.      These  maximum  limits  are  of  interest 

to  the  planner.     For  our  example  the  minimum  number  of  trials,       N       , 

m 

is      2     and  the  maximum  number,       N,.       ,    is      9     . 

We  shall  now  illustrate  the  use  of  the  Table  during;  testing;.     Before 
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starting  the  testing  program,    there  are,    of  course,   no  failures  and  no 

successes  and  we  start  in  the      (0,    0)      position.     The  lower  entry  is 

smaller  than  the  upper,    therefore  a  trial  is  conducted.     If  this  trial  is 

successful,    move  to  the      (0,    1)  position    (0  failures,    1   success);  if  the 

trial  is  a  failure  move  to  the    (1,    0)    position    (1  failure,    0  successes). 

Either  outcome,   however,    leads  to  a  position  within  the  dotted  line  so 

a  second  trial  is  conducted.      The  process  continues  until  we  are  led  to 

a  position  outside  the  dotted  line.      To  get  to  a  position  outside  the  dotted 

line  v/ill  require  at  least  2  trials  and  at  most  9  trials.     Hence,       N        =    2, 
^  m 

and      NM   =   9      . 

To  confirm  these  numbers  note  that  either  2  successive  failures 
or  successes  will  lead  outside  the  line.     Or,    alternate  success  and 
failure  will  lead  one  diagonally  dqwn  the  Table  to  the  (4,    4)  position. 
This  requires      8     trials.     Once  here,   either  another  success  or  failure 
will  lead  one  to  a  position  which  is  outside  the  dotted  line  and  not     shown 
on  the  Table.     Hence,     the  maximum  number  of  trials  is     9      ,   and  is  the 
sum  of  the  largest  row  and  column  designator  plus  one. 

The  testing    program  is  thus  completed  when  experimental  results 

have  led  to  crossing  the  dotted  line  of  the  Table  as  described  above. 

Then  the  value  of     p        is  the  number  of  successes  divided  by  the  number 

e 

of  trials.     A  comparison  of     p        with  the  preas  signed     9     leads  to  the 
Terminal  Decision      d.         (convert)  if      p      >      8;      and  to  the  Terminal 
Decision      d?        (develop),    if      p      <9.      If     p      =9      ,    either  decision 
can  be  made. 

Table      3      (Simple    W)    ,    is  used  in  the  same  manner  as   Table    2    . 
Note  that  the     N'M     is  greater  for  the  Simple    W   .    specifically     N       =   25. 
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CHAPTER      V 
VARIATION  OF   INPUT  PARAMETERS 

Tables  similar  to  Table    2  and  Table    3    may  be  prepared  for  other 
values  of  the  basic  parameters     c     and     8      ,    for  prescribed  Weight 
Functions.     From  these  tables      N^       may  be  determined.     Recall 
from  page    38    that  it  is  only  the  ratio      C/W      ,   which  we  called     c      , 
which  is  a  parameter  of  the  Tables. 

The  curves  in  Figures    17   and    18    show  a  plot  of      N.  .      for  sets  of 
values  of  the  parameters  as  indicated.     Figure    17    is  for  the  Linear      W 
and  Figure    18   for  the  Simple      W    .     These  curves  are  plotted  on  semi- 
log paper.      This  was  done  since  solutions  are  more  easily  obtained 
when     c      is  varied  by  a  factor  of    10   thus  the   semi-log  plot  provides  a 
more  extended  graph.      The  tables,   from  which  the  values  of      N^,      were 
found,    were  obtained  by  means  of  a  CRC  model  102-A  electronic  digital 
computer,    at  the  U.    S     Naval  Postgraduate  School.     Such  Tables  may 
also  be  calculated  with  the  aid  of  the  mathematical  tables  in  Pearson  [6]. 
For  details  of  such  calculations   see  Tucker  [l]. 

Note,   from  Figure  17  for  any  given  value  of     8      ,  as     c     decreases, 
the  number  of  trials  increases.      This  shows  that  if  the  cost  of  testing 
is  decreased  more  trials  are  made  before  a  decision  is  reached.      Con- 
versely,  as  the  cost  of  each  trial  approaches  the  maximum  value  of     W 
(maximum  cost  of  wrong  decision),    fewer  trials  are  conducted.      Or, 
briefly,   the  more  costly  the  testing,   the  fewer  the  number  of  trials. 

Notice,    further,    that  the  number  of  trials  required  is  very  sensitive 
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to  changes  in     3      ,    particularly  when     8     is  close  to  the  value  of     0 
As     8     approaches      0      ,   the  number  of  trials  increases.     A  value  of     8 
close  to     0     requires  the  theory  to  be  more  discriminating  and  this,    in 
turn,    requires  more  trials. 

Figure    18    is   similar  to  Figure    17    .      The  only  difference  being, 
that  for  given  values  of     8     and     c     in  Figure    17,   the  corresponding 
point  in  Figure    18   gives  a  larger  value  of     n     .     In  other  words,    for 
the  same     8     and     c      ,   the  Simple    W   always   results  in  more  trials 
being  required  than  indicated  by  the   Linear    W. 

Summary 

Statistical  Decision  Theory  has  been  applied  to  a   special  problem, 
and  two  different  ways  in  which  the  Weight  Functions  could  be  specified 
have  been  suggested*     A  value  of      N-,       ,    the  maximum  number  of  trials 
required,   is  indicated  by  the  solution.      This  number  can  be  useful  to 
naval  planners  as  a  basis  for  decision  as  to  the  number  of  missiles  to 
be  supplied  to  the  testing  agency  for  the  tests.     It  should  be  noted  that 
the  value  of      N»,      given  by  the  solution  will  have  to  be  increased  to  pro- 
vide for  those  trials  in  which  missiles  are  expended  but,    for  some  rea- 
son,  the  trial  in  invalidated. 

It  is  assumed  that  naval  planners  contemplating  the  use  of  Decision 
Theory  would  have  assistance  from  statisticians  or  mathematicians. 
Solutions  can  be  extremely  time  consuming  to  produce  manually,    hence 
electronic  computers  will  increase  the  practical  usefulness  of  the  theory. 
Dr.    T.    E.    Oberbeck  has  obtained  solutions  for  special  cases  using     sim- 
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pie  and  linear  Weight  Functions,   by  programming  the  problem  for  the 

2 

CRC  model  102-A  computer. 

For  the  use  of  naval  planners,  a  more  complete  set  of  curves  than 
those  given  in  the  chapter  would  facilitate  the  application  of  the  theory. 
Such  a  project  awaits  further  effort  by  research  workers  in  this  field. 

In  cases  where  the  planner  is  unwilling  to  specify  an  a  priori  prob- 
ability density  function  associated  with     p     (see  page  28  )    ,   a  Minimax 
solution  of  the  problem  may  be  obtained.     For  results  on  the  average 
maximum  number  of  trials   see  Breakwell  [3], 


To  be  reported  at  the  Fourth  Annual  Meeting  of  the  Operations  Research 
Society  of  America,   in  May  1956. 
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APPENDIX   A 

The  graphs  for  sequential  tests   shown  by  Figures    6,     7   and    8   of 
Chapter    III  are  constructed  as  follows. 

P 


(1)      The  ratio:   \ 


In 


R  p. 


is  calculated.      Where      n   =    number  of  trials 

s    =:  number  of  successes   (hits) 
q    =    (1    -  p) 


(2)     The  criteria  defining  the  three  regions  are: 

\  /3 

(a)  If  /\  ,<:  |    _^  ,         accept  system    II 

(b)  If  X   R       >      Z  ■■  >         reject  system    II 

(c)  If  T^-—     $     >v    D      <:      ^—        ,  continue  testing 


For   Case   A    :         p        =      .2  p?  =      .  4  C<    ~      .01 

qj      =      .8  q2  =      .6  /S     =      .  1 

then  the  inequality     (a)     is 

0  0-1 


^  1    -e*  1    -«*  9.9 

Solving  for     s      ,   by  logarithms  gives 

s       ^       2.  33      +      .  293n 
Values  of     s     and     n     which  satisfy  this  inequality,   define  th; 
acceptance  region. 
Similarly,    inequality      (b)      is 
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f ) s  (-4-J  >  -4*  ■    -^£    -   'o 

Solving  for     s     gives 

s          ^  -  4. 5$  +       .  -293n 

Values  of     s     and     n  which  satisfy  this  inequality,   define  the 
rejection  region  . 

(3)     The  parallel  lines  in  Figure    6   are  obtained  by  plotting  the   straight 

lines 

s           =  2.33  +        . 293n 

s           =  -4.58  +        . 293n 

In  the  same  manner,    the  straight  lines  in  Figure    7   and    8   are  obtained. 

For  Case    B    :                   s           =  3.05  +        .  293n 

s            =  -4.64  +         . 293n 

For  Case     C    :                  s           =  1.28  +        .  387n 

s            =  -2.51  +        . 387n 
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